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^ nJLf s ^ a y apparatus comprises a displaying 
device for displaying video infonnation for an 
^f^- ■ detecting device for detecting a 
motion of at least a part of the body of the 
observer, and a controlling device for making 
ISffi^" 9 device display a display element 
for input of an operation instruction, "and start- 
ing a process corresponding to * the - display 
element when the detecting device detects a 
specified motion in correspondence to the dis- 
. play . element 
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n^^r,t .MO OF THE INVENTION 

Field of the Invention 

The present invention relate* to a picture record- 

wriftp* to be mounted on the head - 

Related Backoround Art 

Pta « ,sa perspective view to show a cpnstruc 
tioh o?a co^ertiona, HMD-type display appa^ 
The HMD-type display apparatus is otagc^le^pe 
to be mounted on the head of a W^rreS 
obsed of speakers 91u 91* arranged hi correspon- 
STnce to the left and right ears of the « "^J** 
portion 92 set in front of the user's eyes «^~ ns £_ 

^SngTvoice or the like given by the user. 
Her^bSp* U Indicate corres^noe to ^,e 
Sear eye) and the right (ear. eye). Further, the ap- 
lara^'bprovlded ^.«^«*^*-2t 
'K^oorf*-*** for the user to .n P utan.n- 
SS. and the control switch 98 Is connected to 

agreeable 

SSta When the user performs a norm* op- 
e«ttonTuchasdiso.aylngacertein J-BJ-g^ 

=5S=r:5rr2 

sometimes caused the user to lose a sense 
ofwhlTth^^ 

^•C 'oTso through such a control sw,tch. 



ri i M MARY OF THE INVENTION 

An object of the present invention is to provide a 
dlsptay apparatus which is easy 

« 2, instruction of operation and to which the op- 

ttZZXi*"* * ^ p,ex operat,on ,rv 

invenSon tar achieving the above object . character- 

ized by comprising: 



displaying means for displaying video Informa- 

tion for an observer; * ^ ^ f _ f 

detecting means for detecting 'a^obon of at 

.east a part of the body of the *s*™W*^ nQ 
s controlling means for making the ^'splaymg 

meansdisplayadisplayelementforinputofanoper- 

S nSn. and starting a ^j££Z£ 
too to the display element when the detecting means 
detectfa specified motion In correspondence to the 

acte^inthatthedisplayapparatusternc^nte^ 

acte^^th^^ 
« Htenlavina a display elenwht n^najr^^ordance 

25 S^lon detected b^ 

acte ^nthatthecontr^ 

^dSplayapparatus^^ 
ceding sJnd generate 

means detects the specified motion. 

^preferred fc^of thedetecHng rneansiscl^ 
acter£d in that the detect!^ W ^^v 
^ aae taking means for taking in an image of at least a 
JLl 72 body of the^observer, and means for de- 

device and transmits a predetermined sigpai nw» 
fornicating means to the device when the dej 
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tecting means detects the specified motion in cprre- r 
spondence to the device. j? ^ ^; 

A preferred form of the controlling means is char- ; ^ 
acterized in that the controlling means comprises 
means for preliminarily storing features of an object, 5 
and means for comparing the image signal with the 
features stored. 

The display apparatus of the present invention - 
will be described in detail with some embodiments in ■ 
the following description. to 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a perspective view to show the structure 
of the conventional HMD-type display apparatus; is 
Fig. 2 is a perspective view to show the overall 
structure of the display apparatus as an embodi- 
ment erf the present invention; ■■ .-> iV -:,.>.: 
i Flg^ 3 which comprised of Figs. 3A and>3B<is a i>: 
. block diagram to show the structure of the display 20 

• apparatus of Fig. 2; >>r v. *rr.? vr 
Fig. 4 is a flowchart to show the operation of the >*. 
display apparatus of Fig. 2; 

■ Fig. 5 is a drawing to illustrate an example of dis- 
play screen; • --i. ■ ■ y?, -.vr- ■ •.^^■^v^:: 25 

• Fig. 6 is a drawing to illustrate a method vOf |n- -w 
struction operation; v<^?v .:&r!h&4t*& ^ 
Fig. 7 is a drawing to illustrate a method for, reo- 

v ognizing an operation of an operator; v ^^4^ V; 
Vr Figs. 8A and 8Bare drawings to illustrate a spe- 30 
cific example of the method for recognizing an 
operation of an or^raton \ A > 
r Figs; 9A; 9B f and 9C are drawings to illustrate a 
- method for discriminating an.operattoa?instruo- 
/ tion; .v wr. ^-M:^. ^--y^* ■ 35 

Figs. 10A and 1 0B are drawings to illustrate a 
case for discriminating an operation instruction 
ayby;detecting a motion of finger; term,.*. ^ -v^- 
Fig. 11 which comprised of Fig^tlA and 11B is 
a block diagrarrt to,show the stmcrture of the dte- 40 
play apparatus as a second embodiment of the 
present invention; and vv ->■•■:■■. 

Fig. 12 is a drawing to illustrate the principle of a 
visual axis detection method. 

- .7 - ■ 45 
DESCRIPTION OF THE PREFERRED 
EMBODIMENTS , ^ 

Next, the embodiments of the present invention 
will be described with reference to the accompanying so 
drawings. 

Embodiment 1 

Fig. 2 is a perspective view to show the overall 55 
structure of the display apparatus as the first embodi- 
ment of the present invention, Figs.: 3A and;3B are 
block diagrams to show the structure of the display 



apparatus, and Fig. 4 a flowchart to show the opera- 
tion of the display apparatus. ...... . : , . ^ . 

This display apparatus is of t tip HMD (Head 
Mounted Display) type to be mountgjSion the head of 
an operator, which is constructed in a similar arrange- 
ment to the conventional apparatus shown in Fig. 1, 
but which is so arranged that an instruction from the 
operator is input by detecting a motion, for example, 
of the: tip of a finger, thus obviating the control switch 
connected through the wire cable. This display appa- 
ratus is divided roughly into a mount portion 10 which . 
is actually mounted on the head of the operator and 
a processing portion 5 provided separately from the 
mount portion 10. Although the processing portion 5 
can be constructed as incorporated with the mount 
portion 10, it is preferred that the processing portion 
5 be provided separately from the mount portion 10 
in order to decrease the weight of the portion mounted 
on the head of the operator; Typically, the prpcesslng , ■ 
portion 5 is set on a belt portion around the waist of 
the operator. If the display apparatus of the present 
embodiment is used inside a room, the processing 
portion 5 may be installed anywhere in the room. In 
either case, signals etc* are transmitted through radio 
or cable between the mount portion 1 0 and the proc- 
essing portion 5. A battery or the like is used as* 
power supply for the processing portion: 5 antdithe, 
mount portion 10. If the mount portion ? 10 ; ; and the 
processing portion :5 • are connected to each other - ^ 
through a cable, the apparatus may be arranged to 
supply power from the processing portion 5 to the 4 
mount portion 10i: % : Wr^'sr^-'** > :t -'fv 

In the mount portion 10 of the goggle type there ^ 
are provided speakers 1 L , 1 R cx)rresporiding to the left ■ 
and right ears of the operator, displaying portions 2£ 
2r located in front of the left and right eyes of the op- 
erator, for displaying images for the operator, camera 
portions 3t_, 3r for taking in a scene around the oper- 
ator in accordance with the operator's eyes and de* 
tecting a part of the body of the operator, specif ically 
a part of hand (e.g., finger tip 6), and a microphone 4 
for taking in voice given by the operator. Here, the dis- 
playing portions 2l, 2r are constructed as capable of 
displaying a stereoscopic image for the operator and 
each consists, for example, of a liquid crystal display 
member. Also, each camera portion 3 Lf 3 R is set in 
correspondence to the left or right eye, respectively, 
of the operator in the vicinity of each displaying por- 
tion 2l, 2r in order to take in information of an image 
which would be seen through the operator's eyes. 
The two camera portions 3^, 3r constitute a multi-eye 
image taking system. Here, the subscripts L, R indi- 
cate correspondence to the left (ear, eye) and the 
right (ear, eye), respectively. 

The display apparatus of the present embodi- 
ment is arranged in such a manner that the camera 
portions 3l, 3 R detect, for example, a motion or a 
shape of finger tip 6 of the operator and that the entire 
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main body Is controlled in accordance with the motion 
or the shape thus detected or that communication is 
made with an external device to operate the external 
device. 

Next, the detailed block structure of the display 
apparatus Is described referring to Figs. 3A and 3B. 

Each camera portion 3 L , 3r is composed of a lens 
101 L ; 101 R , a lens driving portion 102 L> 102 R for driv- 
ing the lens 1 01 L , 1 01 R to change the focal point or the 
zoom ratio of lens 101 Lf 101 R , an exposure amount 
controlling member 103u, 103 R for adjusting an 
amount of exposure, e.g., aperture or shutter speed, 
a driving portion 104l, 104 R for drive-controlling the 
exposure amount controlling member 103l,103 r and 
a solid state imaging device 105|_, 105^ placed at a 
position where a light image from lens 101 i, 101 R is 
focused after having passed the exposure amount 
controlling member M03u -1Q3r and composed of a 
CCD sensor, for converting the light Image into an 
electric signal. -"■*'■■ - -w^ 1 ?-' ; 

Provided on the output side of each camera por- 
tion 3l,3r is an A/D converter 1 08u 1 08r for convert- 
ing a left or right video signal output from the solid 
state device 1 05|_; 1 05r into a digital signahThe digital 
data from the A/D converters 1 08i.,108 R is arranged 
to be stored in a memory 110. On the output side of 
memory 110 there are a signal processing portion/111 
for effecting luminance signal processing and color 
signal processing, gamma correction processing.'cetc; 
on the video data to convert the video data into data 
suitable for communication or into data suitable for 
display on a monitor, a motion detecting >portton&M2 
for detecting a motion of a part or the whole of a pho- 
tographic screen, and a shape recognizing portion 
1 1 3 for' recognizing a shape or the like of a photo- 
graphed object An output from the signal processing 
portion 111 is put into a communicating portion ; 115 for 
communication with external devices and; also into a 
memory 116 for virtual display as detailed *later^An 
output from the motion detecting portion 112 and an 
output from the shape recognizing portion113 are in- 
put into a data generating portion 114. The data gen- 
erating portion 114, which is for displaying a virtual in- 
dication on the displaying portion 2 L( 2 R , generates 
model data for displaying a virtual fingeror cursor and 
a virtual control panels based on the information 
about the motion of the photographed finger tip. Here, 
the virtual indication means to display on the display- 
ing portion 2 L> 2 R a display object that is not really 
present or that is present but does not exist at a cer- 
tain place in a three-dimensional space, as if it is ac- 
tually present at the place. An output from the data 
generating portion 114 is input into a memory;117 for 
indication on parent-child screens as described -be- 
low. "'• * 

The display apparatus of the present embodi- 
ment is arranged to display a parent screen and a 
child screen superimposed inside the parent screen 



on the displaying portions 2 L , 2 R . The parent screen 
shows a surrounding scene around the operator as 
taken by the cameras 3l, 3 R . On the otbpr hand, the 
child screen is a screen for the display^fepparatus to 
5 indicate, independently of the surrounding scene, irv 
formation of a preliminarily photographed image or 
the like, which may be a moving picture or a still pic- 
ture. For displaying such parent screen and child 
screen and for performing the above-described vir- 
10 tual display, the display apparatuses provided with 
the imaginary display memory 116 for displaying a 
virtual indication, i.e., a virtual f inger or curs<*, otfa ; 
virtual control panel, and the parent-child screen dis- 
play memory 117 for temporarily storing image data 
15 to be displayed on the parent screen and the child 
screen. Further, there is provided a display process- 
ing pbrtk>n118 into which image data stored In the vir- 
tual display memory 116 and parent-child Image dis- 
play m^irmiV 117 is input, by which images for the 
20 parent screen.ivthe child screen; ^and the virtual dls- ; 
play screen are combined, and by which an image is 
separated with necessity into images for the left and 
right displaying portions 2^ ^. The dismay process- 
ing portion 118 outputs signals ima predetermined 
25 signal form for display (NTSC, RGB, eto^-Furtheiv 
the apparatus is provided with a display controller 119 
for controlling display timing, and a timing signal from; 
the display controller is supplied to the virtual display y 
' memory 116, the parent-child screen display wrnipry^ ^ 
30 117; the display processing portion 11 8, and ^driver* • 
122, The driver 122 is for driving the left and right dis- 
' playing portions 2 L , 2 R . Driven by the driver 122, the 
left and right displaying portions 2t, 2r displayihe left 
and right screens, respectively, in accbfdance with 
-35 image signals from the display processing ^portion 
118. ^ . •'* j " • : '^ y: ■^P^'- ' 

For integrally controlling the above-described 
portions, a whole controlling portion 123>is 'provided 
in the processing portion 5 (Fig. 2); The whole control- 
40 ling portion 123 also executes an image taking con- 
trol such as lens drive, display control; and process- 
ing according to each application. Further/<the micro- 
phone 4 and the speakers 1l. 1 r provided in the 
mount portion 10 are also connected to Hhe> whole 
45 controlling portion 123. The apparatus is f urther pro- 
vided with a power supply 124 for supplying the pb^er 
to the entire apparatus. The power supply 124te »ri- 
structed specifically of a battery. Next, the operation 
of the display apparatus is described referring to Fig. 

so 4. • " J k - • ' v 

First, the exposure amount control and focal point 
control are executed (step 151). The exposure 
amount control is made by conducting such a-feed* 
back control that the imaging devices 105 L , 105 R take 

55 in images while driving the exposure amount control- 
ling members 103 Lt 103 R , the images are converted 
into digital values by the A/D converters 108i;,i08 Rf 
the digital values are temporarily stored in the mem- 
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ory 110 f and the exposure amount controlling mem- 
bers 1 03l, 1 03 R are again driven so that a level of data 
in a predetermined range of screen (corresponding to 
a luminance level of subject) becomes a suitable lev* 
el. Similarly, the focal point control is performed in 
such a manner that as to the data temporarily stored 
in the memory 110, data in a predetermined range of 
screen is let to pass a high-pass filter to measure a 
level of high-frequency components and the lenses 
101 Lf 101 R are driven so as to make this level maxi- 
mum. Subsequently, a motion and a shape of an ob- 
ject are recognized from the photographed screen by 
the motion detecting portion 112 and the shape rec- 
ognizing portion 113 (step 152). There are a variety 
of methods for recognizing a motion or a shape, 
among which a simple example is a method (template 
matching) in which data of an image to be a reference 
is prepared and a correlation is obtained between the 
reference image data and an image to be recognized. 
The processes performed at these steps 151; 152 are 
well known as such processes as: photometry, dis- 
tance measurement, and camera shake correction in 
ordinary video cameras, and, therefore, detailed de- 
scription thereof is omitted herein. i~ 

; Next, it is determined whether the recognized ob- 
ject is a specified object, for example, a finger (step 
153). If it is not the specified object, ; th&f low returns 
to the above step 151 in order to await recognition of 
the specified object If the recognized object is deter- 
mined as the specific object, it is checked whether the 
specified object lies within a specified area in the 
space (step 154). A reason to check if it lies within the 
specif ied area is to determine if the operator moved 
a finger or the like into the specified area in order to 
give an instruction of operation. If the specif ied object 
is absent in the specif ied area, the flow returns to 
step 151 similarly as described above. If it is within 
the specified area, it is then determined if the speci- 
fied object conducted a specif ied motion (step 155). 
The present embodiment is so arranged that an in- 
struction from the operator is input through a motion 
of the operator, for example, which a pointing motion, 
a hand opening motion, or a hand closing motion is 
made for example by a finger or fingers. Here it is de- 
termined if either one of such specified motions was 
performed. If it is not one of the specified motions, the 
flow returns to step 151. 

If the motion is determined as a specified motion 
at step 155, a process corresponding to the motion is 
next carried out(step 1 56). The process herein means 
execution of an application designated by the opera- 
tor's motion (for example, calling or execution of a fac- 
simile function). In response to the specified opera- 
tion of the operator, the apparatus changes icons or 
the shape, size, or color of the virtual switch panel as 
displayed as a virtual indication, and if necessary, a 
sound is further output through the speakers 1 L ,>1 R 
(step 157). The change of the icons or the shape of 



the virtual switch panel is made by producing corre- 
sponding display data by the data generating portion 
114. Also, a screen corresponding'tb^the designated 
application is called out from the^kxternal device 

5 through the communicating portion 115 or from an un- 
represented memory portion, is temporarily stored in 
the imaginary display memory 116 or the parent-child 
screen display memory 117, and is displayed on the 
left and right displaying portions 2 L , 2r through the 

io display processing portion 118. Control of a display 
timing signal and reading out of the memory on this 
occasion is executed by the display controller 119. 

Next, it is determined whether or not interruption 
of power supply should be made (step 158). For ex- 

15 ample, if no operation has been done over a prede- 
termined time, the apparatus determines that the 
power supply should be interrupted. If the apparatus 
determines that the power supply should be main- 
tained, the flow returns to step 151; if the power sup- 

20 ply should be interrupted, the power supply is actually 
interrupted (step 159) and the overall operation is 
ended. 

Next described are display contents presented 
for the operator through the displaying portions 2^ 2r. 

25 First described referring to Fig. 5 is a case of perform- 
ing the operation of the entire main body. Here, the 
displaying portions 2u- 2r correspond to the left and 
right eyes of the operator, and a stereoscopic image 
display is made by these displaying portions 2* 2* r 

30 but the description herein is given as regarding ster- 
eoscopically displayed images as a single image. 

With the mount portion 1 0 mounted, a parent im- 
age 22 is displayed corresponding to nearly the whole 
of the field while a child image 23 smaller than the 

35 parent image is displayed in the parent image 22. Fur- 
ther, an operator's hand 24 detected in the above 
manner is also displayed as a virtual indication at a 
position apparently closer to the operator than the 
parent image 22. The hand 24 may be displayed as a 

40 virtual indication in a specific form of hand or in a 
shape of arrow or cursor. A control switch panel 27 is 
also displayed as a virtual indication similarly as the 
hand 24 of virtual indication. This control switch panel 
27 is an example of display of control switches 25, 26 

45 for frame return and frame feed of child screen dis- 
play, and these control switches 25, 26 are also dis- 
played as a virtual indication. 

Here, the parent screen 22 shows a scene around 
the operator as taken in by the left and right camera 

so portions 3^ 3 R . The child screen 23 shows a prelimi- 
narily taken picture (a car herein). The picture on the 
child screen 23 may be a moving picture or a still pic- 
ture, which may be read out of a memory device pro- 
vided in the processing portion 5 or may be captured 

55 by communication from the outside through the com- 
municating portion 115. 

Here, let us consider a case that the operator in- 
tends to change the display contents on the child 
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screen 23. Let us assume that a still image is dis- 
played on the child screen 23 and an operator's finger 
is selected as the specified object in the above de- 
scription. If the operator desires to display a next 
screen or if the operator wants to feed a frame, the 
operator first locates a finger thereof within the field 
of the camera portions 3 Lf 3 R . Then the operator 
moves the finger to the position where the control 
switch panel 27 is displayed as a virtual indication 
(th#|?bsHion where it appears existing there), and 
performs an operation to push the control switch 26 
with the finger. This pushing operation with the f inger 
is taken by the camera portions 3t, 3 R and this oper- 
atibri is recognized according to the above-described 
processing, whereby, based on the control from the 
wfete controlling portion 123, the next screen is read 
out and displayed on the child screen 23. Similarly, if 
the control switch 25 is pressed on a virtual basts, the 
child soBert 23 is transferred to a screen one before. 
vN^ly^althou^ 

ncrt present in the actual space; by performing the 
pushing operation with the finger in the real space 
a^f^hdjng to the virtue 

pnx»ss^fe earned but based on the performance of 

' the^^ibri.^^:; V V- 
^t>Aere1^he apparat^^ 

atfaMi^ffiertt thatif rio^peration is performed for 
^ce^ttlime switch 
f I^riel27u^^ ? 
27:itui™^ 

to th^f^edfied positbn^pr perfonns a 
tkih^^thev^pparatuis : n^ibe? arrahgednto de^ it 
th^Hi^ 

pla^^virfiial control switch panel 57. r ^ 
A^fte&d^ 

troi devices set outside the display apparatus* In this 
ei&mpieiifacsimileiahd telephone «re assumed as 
thS^exterrial devices and a memo pad is further pro- 
vided bn-a virtual basis." * - *'i " m 

The displaying portions 2l, 2« show virtual indi- 
cations of icon 28 indicating the memo pad, icon 29 
indicating telephone, and icon 30 indicating facsimile. 
Here, the icons can be considered as equivalent to 
pictorial symbols used in graphical user interface 
(GUI) for personal computers etc. Also, display con- 
tents on the parent screen 22 and on the child screen 
23 are assumed to be thesame as those shown in Fig. 

5;. ' - ; ■ " " 

Let us consider here a case that the operator 
wants data transfer through facsimile. Then the oper- 
ator points out the icon 30 of facsimile out of the icons 
28 to 30, and dicks it (an action to press it). This click 
action is detected, so that a virtual switch panel (not 
shown) for facsimile transmission is displayed on a 
virtual basis. Thus, the operator can manipulate vir- 
tual switches in accordance with instructions on the 
panel. In this case, the apparatus may be constructed 
in such an arrangement that once it is detected that 



the icon is clicked, the operator is informed of recep- 
tion of the operation by changing the shape, size, or 
color of the icon (affirmation display qfjeception of 
the icon operation) or generating a soujjff through the 

5 speakers 1f 1 R . 

After completion of the operation for facsimile 
transmission, the icon 30 of facsimile and the other 
icons may be erased from on the screen or may be 
moved to a location where they do not hinder the field, 
10 for example to a corner of the screen. Alternatively, 
the size of the icons may be decreased. The appara- 
tus may be arranged so that the icons or the virtual 
switch panel is again displayed when the operator 
moves the finger to a specif led position or performs 
15 a specif ied action and the camera portions 3 L , 3* de- 
tect it 

The abbve description concerned the example or 
facsimile trahsmissbn, but the same is applicable to 
the c»sb of maldng^a Iphone call. Briefly descrtbihg; 
20 when the operator pkjrtorirhs an action of clicking the 
vHual icon 29 of telephone; the whole 
tion 123 calls a task to process a telephone function 
outside orthi^ 

microphone 4 and speakers 1 r ,1l with the telephone 
25 function^ This enables the operator to communicate 
by voice ^with a calling f partner through the communi- 
cating portion 115 arid through a telephone set sep- 
arately from the display apparatus or a telephone of 
; calling destination: ^ receptfoh coiv 

30 firmatibh indication of icon operation and display 
change of icons after end of communication is thb 
same 4 as in the example of facsimile as described 



above. 



35 



A case of the memo pad is briefly described, Siro^ 
ilarly as in the examples of facsimile and telephone* ; 
when the virtual icon 28 of memo pad Is clicked by fin- 
ger, a virtual note for memorandum is displayed^ 
the screen. When the operator takes an actton dl writ- / 
ing a picture or letters on the virtual note with fing^; 
40 a motion of the f inger is detected and the detects* 
motion is processed by the whole processing portion 
23 or an external device to be stored or to be subject- 
ed to character recognition. Namely, a function of pen 
input computer can be realized. In this case; a touch 
45 panel in the pen input computer is displayed on a vir- 
tual basis by the displaying portions 2i_, 2*. - ' 'i 
Next described referring to Figs. 7, 8A, and 8B Is 
how to detect an action against the virtually displayed 
icons or switch panel, 
so In Fig. 7, a display area 44 of the display appara- 

tus is shown by the frame of solid line. Detection 
areas 41-43 are set for the respective icons 28-30 of 
memo pad, telephone, and facsimile, and the detec- 
tion areas 41-43 are represented by the dashed lines. 
55 When the camera portions 3l, 3r detect a finger, 

it is checked where the finger lies within the display 
area 44. In the illustrated example, the finger lies In 
the detection area 42 of icon 29 of telephone, so that 

6 
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it is regarded that the operator pointed out the. tele- 
phone. The judgment of whether the finger lies within 
the detection area 42 for icon 29 of telephone may be 
done by taking a correlation with some images, as 
shown in Figs. 8A and 8B. Fig. 8A shows a case in 
which the finger is still outside the detection area 42, 
while Fig. 8B shows an image where the finger is lo- 
cated inside the detection area 42. Accordingly, the 
whole processing unit 123 may be arranged in such 
a manner that it preliminarily stores the image of Fig. 
8A, continuously takes in images from the camera 
portions 3 L , 3 R , and detects any change from the im- 
age of Fig. 8A. In order to prevent erroneous recogni- 
tion due to an object other than the finger entering the 
detection area 42, the processing portion may be ar- 
ranged to preliminarily store the image with the finger 
therein as in Fig. 8B and to determine that the finger 
is in the detection area if a correlation with the image 
of Fig. 8B is more than a specif ic value. --^ v 

Next described referring to Figa; 9A; 9By and 9C 
is how to detect whether or not an action of pressing 
(or clicking) the virtually displayed icon with finger is 
taken. 7 Figs. 9A, 9B, and 9C are drawings to show re- 
sults obtained by talcing images of fingers of operator 
and extracting features such as contours from the im- 
ages. Namely, Fig. 9A shows an action of pointing out 
with .finger. Fig. 9B a state of an opening palm, and 
Fig. 9C a state of a holding hand. .,.-,:\. .> ' v ^-- . 

•...;^rThe camera portions 3 l ,3r take images oftheop-/: 
orator's fingers; and what form the fingers are taking 
is determined by obtaining correlations by reference 
to the images shown in Figs. 9Ato 9C. Of course, im- 
ages other than the images shown in Figs. 9A to 9C 
may be employed, or still more forms (reference im- 
ages) may be prepared. Then it is determined that; for 
example, when the fingers in the form of Fig. 9Aenter 
either frame of detection area 41-43, the outstanding, 
icon is selected. Then, if the fingers change the form 
through the form of Fig. 9B into the form of Fig. 9C 
within a predetermined time, it is determined that exe- 
cution of processing indicated by the outstanding icon 
is instructed; or, operation contents can be deter- 
mined from a motion of fingers. 

A method for detecting the finger motion is next 
described referring to Figs. 1 0A and 10B. 

Suppose Fig. 1 0A shows an image of fingers tak- 
en at a certain time and Fig. 10B an image of fingers 
taken after a lapse of a predetermined time there- 
from. Amotion and a speed of fingers can be obtained 
from these two images. Accordingly, if the fingers go 
at a speed over a predetermined speed into a frame 
of detection area 41-43, it can be determined that the 
outstanding icon is selected and execution is instruct- 
ed thereby. 

Incidentally, because in the present embodiment 
the camera portions 3 Ll 3 R are provided in the mount 
. portion 10 of the HMD type display apparatus, they 
are mounted on the head of the operator after all. In 



this case, the operator's moving the head results in 
erroneously detecting the speed of fingers. It is thus 
desirable that the speed of fingers' tfjiich becomes a 
reference for determining instruction of icon and exe- 

s cution thereof, be set sufficiently higher than a mov- 
ing speed of the head. Alternatively, because the 
camera portions 3 L , 3 R are also taking a scene around 
the operator other than the fingers, the motion of fin- 
gers may be obtained by calculating a relative speed 

10 between the motion of image of the surrounding 
scene and the motion of fingers. 

The methods for obtaining correlations between 
a plurality of images and for detecting a motion of an 
object in an image as described above are substan- 

15 tially the same as the camera shake detection and 
camera shake correction technology as employed in 
ordinary video cameras, which are readily -under- 
stood by those skilled in the art, and therefore, more 
detailed description is omitted herein. 

20 '.>..-•■ •>;.:• 

Embodiment 2 

The display apparatus of the first embodiment as 
described above was so arranged that an instruction 

25 from the operator was input by detecting a motion of 
a part, for example the finger tip, of the operator's 
body, but the apparatus may be constructed in such 
an arrangement that in addition to the arrangement 
for detecting the motion of the finger tip etc. or instead 

30 of this arrangement, a means for detecting the visual 
axes of the observer is provided, whereby the instruc- 
tion from the operator is input by detecting the visual 
axis of the observer. The following description con- 
cerns an embodiment in which an instruc*k>nfrom the 

35 operator is input by detecting the visual axes of the 
observer. 

- Figs. 11Aand 11 B are block diagrams to show the 
structure of the display apparatus as a second em- 
bodiment of the present invention. 

40 This display apparatus is characterized in that an 

instruction from the operator is attained by detecting 
the visual axes of the operator, and is constructed 
substantially in the same structure as the display ap- 
paratus of the first embodiment as described above 

45 except that visual axis detecting means therefor (vis- 
ual axis detecting portions and gazing point position 
processing portions as detailed later) is provided. 
Since similar constituent portions to those in the dis- 
play apparatus of the first embodiment have similar 

50 operations, they are denoted by same reference nu- 
merals in Figs. 11 A and 11B and description thereof 
is omitted herein. 

The visual axis detecting means for detecting the 
visual axes of the observer is composed of visual axis 

55 detecting portions 1 30 L , 1 30 R for detecting the visual 
axes of the left and right eyes, respectively, of the op- 
erator, and gazing point position processing portions 
131 L , 131 R for determining where the left and right 
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eyes of the operator are gazing on the respective, vir- 
tual screens displayed on the displaying portions 2l, 
2 R , based on the visual axes of the left and right eyes 
detected by the visual axis detecting portions 130 L , 
1 30 R , and generating a gazing point signal for indicat- 
ing the gazing position as a gazing point over images 
on the displaying portions 2 L , 2 R . 

Each gazing point signal generated by each gaz- 
ing point position processing portion 131 L , 131 R is in- 
put into the data generating portion 114. In addition to 
the processing for generating model data for display- 
ing the virtual fingers or the virtual control panel, 
based on the information of a motion of photographed 
finger tips, the data generating portion 114 performs 
processing for generating model data for displaying a 
virtual gazing point, based on the gazing, point signals 
input: from the respective gazing point position proc- 
essing portions 131 L , 131 R . Similar processing is car- 
ried out as in the case of the first embodiment as de- 
scribed above, based on the model data generated by 
the data generating portion 114, whereby the gazing 
point is displayed on the displaying portion 2l, 2r. 

w In the display apparatus of the present embodi- 
ment selection of icon or control switch panel is made 
using' the gazing point displayed on the displaying 
pcrtidh 2L f ^. Namely, selection is made when the 
operator is gazing at one desired to execute out of the 
it^br switch panels displayed. In the display appar 
ratus. execution of a selected icon or the like is donfc 
when the operator gazes at an icon for a predeter- 
mined time or when the operator takes a specified 
motion (for example, a motion of making a predeter- 
mined shape of fingers or hand being displayed to- 
gether with the gazing point recognized; or also uses 
a mechanical switch). ^ ^ 

Next described is a process for detecting a direc- 
tion of the visual axis of the left or right eye of the op- 
erator^ ■ v ' 

Fig. 12 is a drawing for illustrating the principle of 
the visual axis detecting method. This visual axis de- 
tecting method is detailed in Japanese Laid-open Pa- 
tent Application No. 5-68188. The visual axis detect- 
ing method as disclosed in the application is briefly 
described in the following. 

In Fig. 12, reference numeral 300 designates an 
eyeball, 301 the cornea, and 303 the iris. The block 
surrounded by the chain line represents the visual 
axisdetecting portion (130 L , 130r) shown in Figs. 11A 
and 11B. This visual axis detecting portion (130l, 
130r) is composed of a light source 161 such as a 
light-emitting diode (LED) for emitting infrared light, 
a light-projecting lens 162 and a half mirror 163 ar- 
ranged in order for guiding a light beam emitted from 
the light source 3 61 to the cornea 301, and a light- 
receiving lens 164 and a photoelectric device 165 
such as CCD arranged in order along the traveling di- 
rection of the light beam reflected by the cornea 301 
and passing through the half mirror 1 63. Then an out- 



put from the photoelectric device 165 forming the vis- 
ual axis detecting portion (130 L , 130r) is input into the 
gazing point position processing portionrM31 L , 131 r). 
The infrared light emitted from th^fyight source 
5 161 is converted into a parallel beam by the light- 
projecting lens 1 62, and the parallel beam is reflected 
by the half mirror 163 to illuminate the cornea 301 in 
the eyeball 300. Part of the infrared light illuminating 
the cornea 301 is reflected by the surface of the cor- 
10 nea 301. The reflected light from the surface of the 
cornea 301 passes through the half mirror 163 to be 
converged by the light-receiving lens 164 and theriip 
be focused on the image plane of the photoelectric > 
device 165. At this moment, a corneal reflection^ 
is age d generated by part of the infrared light reflected! ■ : 
by the surface of the cornea 301 is formed at position ^ 
Zdd on the image plane of the photoelec^cdeylc&S 
165. and reflection images a. b of the both edges of > 
the iris 303 generated by part of infrared light r^ebfc^ 
26 ed thereby aire formed at positions Ziaa and Zbb; <: 

A rotational angle eb (visual axis direction) tftl^ p : . 
eyeball axis B of the eyeball 300 can be obtarnerff rbjjn 
the position Zdd of the corneal refledk>rVim^e^tfn»>^ 
formed and the positions Zaa, Zbb of the reflieictk^ ?^ 
25 images a, b of the both edges of the iris 303 rel^iye ^ 
to the optical axis Aon the image plane of the pho^-^ 
electric device 1 65; Namely, in the gazing |x>intpo^ ^ 
ition processing portion (131 L , 131r), the rotatipffti ^ 
angle 6b of the eyeball axis B can be obtained f. 
30 tecting a" positional-, relation -betweeri*e:opflc^5ia^:%-: 
A and each position Zaa, Zbb, Zdd on the image ^ane 
thereof from outputs from the photoelectric: df^-^ 
165 and obtaining a magnification of the dlgltt- 
recehnng lens 1 64 from a distance L1 between this ib^ 
35 flective surface of the cornea 301 and the^ight- v 
receiving lens 164 and a distance L0 between the 
light-receiving lens 164 and the image plane of * the 
phbtoelectric device 165. - -i 

The operation of this display apparatus is brief 
40 described in the following. ; l : a- 

The mount portion 1 0 is first mounted. On this oc- 
casion, the images shown in Fig. 5 (the images of vir- 
tual indications of the operator's fingers and the con- 
trol switch panel 27) are displayed on the displaying 
45 portion 2l, 2 r , and a virtual indication of the gazing 
point is given on the thus displayed images and in the 
direction of the visual axis of the operator. ^ 

Now, when the operator is gazing at either one =of 
the control switches 25, 26 for frame return and 
50 frame feed in the control switch panel 27 displayed on 
the displaying portions 2l, 2 r , the gazing point dis- 
played on the images moves in synchronization with 
movement of the operator's visual axes, and the gaz- 
ing point is displayed at a gazing position on the con- 
55 trol switch. Execution of processing related to the 
gazed control switch is recognized by continuing gaz- 
ing fora predetermined time or by the operatorfs'per- 
forming a specified motion, whereby, based on coiv 

8 
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trol from the whole controlling portion 123, the proc- 
essing related to the control switch recognized as to 
be executed (frame return or frame feed of screen) is 
executed. 

As described above, the display apparatus of the s 
present embodiment enables input of instruction from 
the operator with the visual axes by replacing the pos- 
ition of the operator's fingers displayed as in the dis- 
play apparatus of the first embodiment described pre- 
viously, with the gazing point obtained from the visual 10 
axis direction of the operator. 

Although the present embodiment was described 
as an example in which the position at which the op- 
erator was gazing was displayed as superimposed on 
the images on the displaying portion 2l, 2 r , selection 15 
of a control switch eta and execution thereof can be 
made by similar processing without displaying the 
gazing point 

; ; . Further, the present embodiment activates, with 
the visual axes, execution of the processing related to 20 
the control switch eta by continuing gazing for a pre- 
determined time, but the invention is by no means lim- 
ited to it; for example, the execution can be activated, 
for example, when a specified motion of the visual 
axes of the operator is detected. 25 

In the above-described display apparatus, the 
operability of apparatus can be further improved by 
providing the apparatus with an adjusting function for 
absorbing differences of visual axis detecting proper- 
ties between individuals or providing the apparatus 30 
with a process to store data in a memory.: 

Other Embodiments 

For the display apparatus of the present invert- 35 
tion, a variety of embodiments can be considered 
other than the above-described embodiments. For 
example, though the camera portions were incorpo- 
rated with the displaying portions of the HMD type 
main body in the above embodiments, the camera ao 
portions may be arranged as separate therefrom. In 
the case of the separate arrangement, for example, if 
the cameras are installed at a certain place in the 
room so as to take images of the whole of operator, 
there is an advantage that the whole motion of the 45 
body of the operator can be detected as well as the 
fingers. However, it becomes difficult to detect a fine 
motion of a finger or the like. Further, because the 
field of the cameras is not coincident with the opera- 
tor's field, this arrangement is not suitable for appli- so 
cations requiring an indication of the operator's field 
on the displaying portions. 

Also, the apparatus can be constructed in such 
an arrangement that the displaying portions are not 
mounted on the head of the operator. In this case, a 55 
monitor is set in the room like a normal television 
monitor and a camera takes in an instruction of the 
operator. In this arrangement, the operator can con- 



trol channels of television or a video system, or can 
control a computer, in accordance with the taken in- 
struction. Employed for the focusing operation of 
camera portions and the control of j^cposure amount 
are well known controls used in silver-salt cameras or 
video cameras. 

The above-described embodiments were ar- 
ranged with two camera portions for the left and right 
eyes, but the apparatus may be constructed using 
only one camera portion. In the case of the double- 
eye arrangement an advantage of obtaining stereo- 
scopic information (distance, shape) can be enjoyed; 
but, if it is not necessary, the apparatus may be con- 
structed in a single-eye structure, whereby the 
weight, the size, and the cost can be reduced. 

Among the HMD (Head Mounted Display) type 
display apparatus as described above, there are 
those of a type in which the displaying portions are 
made from a transparent (optically see-through) 
member whereby the operator can see a scene in the 
external field therethrough in a normal state; and with 
necessity, display contents of liquid crystal or the like 
are indicated on an optical surface in front of trie op- 
erator (apparatus of a so-called head up display 
type). In the case of using such apparatus, a mark 
can be displayed over a portion of facsimile actually 
seen through the displaying apparatus for example, 
thereby clearly showing the object to be instructed. 

As for the virtual switch panel or the icons dis^ 
played as virtual indications using the displaying por- 
tions, various changes can be made according to ap- 
plications with regard to displaying positions, display- 
ing timing, erasing timing, color, shape, and genera- 
tion of sound. ^: ! .2m.^ 

As the object for detection of operation {specified 
object), any other object can be used than the fingers 
or the visual axes as in the above examples. For ex- 
ample, an object with a feature (a specific form or col- ' 
or, a pen having a temperature, or the like) may be 
used to facilitate recognition. Further, without recog- 
nition as a specified object, the apparatus may be ar- 
ranged to judge an object detected in a specified area 
of screen, at a specified time, and in a specified ap- 
plication as an object for instruction (instructing ob- 
ject). In this case, the recognition processing is sim- 
plified. Also, once the instructing object or specif ied 
object is detected, the apparatus may be arranged to 
follow it This arrangement realizes high-speed proc- 
essing because it does not have to continuously per- 
form the object recognition. 

The operated object was facsimile or telephone 
in the above embodiments, but the invention is not 
limited to these; for example, the invention can be ap- 
plied to air conditioners, interphones, illuminations, 
and a variety of devices used inside or outside by pre- 
liminarily registering them in the main body as to be 
recognized. The icons displayed on a virtual basis are 
not limited to icons preliminarily registered in the main 
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body as in ordinary computers, but an object actually 
imaged through a camera can be displayed as it is. 
For example, if a facsimile device is set in a room and 
thd operator turns the eyes toward it, the camera por- 
tions take an image of facsimile, so that the facsimile 
is displayed on the displaying portions. Then the ap- 
paratus may be arranged to perform processing relat- 
ed to the facsimile by the operator's pointing it out 
similarly as the icon. On this occasion, the color of the 
image of facsimile thus recognized may be arranged 
to be%hanged in order to indicate that the facsimile 
is recognized. 

fi^TKe above embodiments were so arranged that 
detection of operation instruction was done by detect- 
ing the operator's operation, for Example moving a 
finger with respect to an object of an image displayed 
by the' displaying portibhs/ but a mechanical switch 
zriap also v be ■ use<L ? Especially,- use of mechanical 

power supply to 

Hri^rra^ " ; >^ 'r&l 

# ^^^he^ tt^ 

•so arranged that the jixi^^ 

#nfro&time/butt^^^ 

^nMhaye to be int^ all portions of the main 

4bo^^ 

example ^uch thatithe! power Supply fe lritemjpted 
•only teethe cam the displaying 

'^ortbtei Partk^ if there are* plurality of canrv- 
>eli^i^ only one of 

- tfii^his always driven arid with detection of a specif ic 
^iTOt^iidtion of operator tHrobgh the camera; the power 
' supply iststarted to the other camera portions with 
necessity. In another case, the powerdissipation may 
tbiir substantially decreased; without completely inter- 
rupting the power supply, by lowering an internal exe- 
cution speed of GPU etc. --\ 

Claims V ; 

f1 A display apparatus comprising: 

playing means for displaying image infor- 
mation for an observer; 
^ r detecting means for detecting a motion of 
- " r at least a part of the body of said observer; and 
u ^.controlling means for making saiddisplay- 
ihg means display a display element for input of 
: an operation instruction, and starting a process 
corresponding to said display element when said 
detecting means detects a specified motion in 
correspondence to said display element 

2. The apparatus according to* Claim 1, wherein 
said display apparatus is mounted on the head of 
said observer. 

3. The apparatus according to Claim 1, wherein 



said detecting means has visual axis detecting 
means for detecting a visual axis of the observer: 

4. The apparatus according to CI^A 3, wherein 
5 said controlling means has means for comparing 

a direction corresponding to said display element 
with a direction of the visual axis detected by said 
visual axis detecting means. 

10 5. The apparatus according to Claim 1 , wherein 
said controlling means has means for displaying 
a display element moviiig In accordance with the 
motion detected by said detecting means, on said 
displaying means. ? 

15 ' • • "" ; ' : " v '•.f : - : "^;^" 

6. The apparatus according to Claim 5, wherein - 
said controlling means has means for changing a v 
display property of said display element inrwvlngf 

^ ^in accordance with ^ 
20 -v-: ^idetectirig means? v-> v ^- ■ *&<&.-jfm*^ 

7. The apparatus according to Claim 1 .further ODit^^ 
Uprising sound ^ge^ratinS means for gOTerating^ 

• •! - f ^sound; whereinrsaid controlling mearisvm akesr 
25 said sound generating rtieans function when saW 

/detWirig meahs ? detects specified motion. 
„■ . \ - = - i ■ - - ? - • ; -■.>. . -W ■ -- « - v - y- ■ V-' : ; &r v ■ 

8. The apparatus according ^foVCIaim^l, whereiiv 
v " <: said ■ detecting means ^comprises s image • taidrig' 

30 ^-means for taking in an image of at least a^part of 
the body of said observer; and mean* for detect- 
ing a motion of the at least part of the body of said 
observer from an image signal from^aid irrlage 
taking means. 

35 ■ ■ v -' 1 - • • ;: ' °" * " •"• 

9. The apparatus according to Claims, wherein 
^ said image taking means has a plurality of image 

f taking means with parallax. -.^-^'v-; 

40 10. The apparatus according to Claim 8, wherein 
said detecting means has means for detecting a 
hand or a finger of said observer within a spatial 
range preliminarily determined. 

45 11.- The apparatus according to Claim 1, comprising 
: communicating means for transmitting a signal to 
a device installed outside the apparatus, wherein 
said controlling means makes said displaying 
means display a display element corresponding 

so to said device, and transmits a predetermined 

signal through said communicating means to said 
device when said detecting means detects the 
specified motion in correspondence to said de- 
vice. 

55 

12. The apparatus according to Claim 8, wherein 
vsaid controlling means comprises means for pre- 
liminarily storing features of an object, and 
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means for comparing said image signal with the 
features stored. 

13. The apparatus according to Claim 9, wherein, if 
said detecting means do not detect motion of at s 
least one portion of body of said observer for a 
predetermined time or more, then one of said 
plurality of image taking means is excluded to 
stop the motion of the one thereof. 



14. The apparatus according to Claim 13, wherein, if 
the one of said plurality of image taking means 
detects said motion, then motions of the other im- 
age taking means restart 



16. A display apparatus according to claim 15, which 
is arranged to display a representation of the 
movement, positioning, orientation or configura- 
tion of the objector part of a body. v-VV^V: 
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15. A display apparatus in which movement, position- 
ing, orientation or configuration of an object or 
part of a body, for example a human body, is de- 
tected remotely, for example using optical detec- 
tion means, and is referenced to an element dis- 20 
played by the display apparatus so as to cause an 
operation to be carried out when a determined re- 
lationship exists between the display element 
and the object of part of a body. 
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